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ABSTRACT 

This paper reports the findings of a study which seeks to identify the information and communications technology (ICT) 
literacy levels of trainee teachers, by investigating their ICT proficiency using a task-bask assessment instrument. The 
Delphi technique was used as a primary validation method for the new assessment tool and the ICT literacy task-based 
assessment tool was developed to assess trainee teacher’s levels of ICT-literacy. The findings showed that the 
ICT-literacy task-based assessment instrument were able to predict trainee teacher's ICT-literacy area of weaknesses. 
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1. INTRODUCTION 

The central concern of this study is to develop an ICT-literacy task-based assessment (TBA) instrument that 
may be used to evaluate trainee teacher’s level of ICT-literacy. The current literature acknowledges the need 
for a measurement instrument that evaluates level of ICT-literacy levels (Wong 2002; Markauskaite 2005; 
Katz & Macklin 2007). 

This type of measurement instrument is currently used as an entry level testing tool for university and job 
placements. However, existing ICT-literacy assessment instruments are either too expensive to be 
implemented, or too rigid with their expected answers. Moreover, they are not tailored to teacher’s individual 
needs. The existing instruments either uses self-efficacy techniques or step-by-step task/instructions whereby 
they do not allow flexibility and creativity in completing the task (International ICT Literacy Panel 2002; 
Cox & Marshall 2007). 

Following the inauguration of the International ICT-literacy Panel in 2000, the International Society for 
Technology in Education (ISTE) in 2008, proposed that in order to accurately assess level of ICT-literacy, 
such an assessment must include both technical and information literacy (International ICT Literacy Panel 
2002; International Society for Technology in Education 2008). Two ICT standards which are the 
Information Literacy Competency Standards for Higher Education developed by the Association of College 
and Research Libraries (ACRL) and the Australian and New Zealand Information Literacy (ANZIIL) also 
echo the views of the 2000 International ICT-literacy Panel and 2008 ISTE (Association of College and 
Research Libraries 2000; ANZIIL 2008). 


2. WHY TRAINEE TEACHERS? 

Currently, ICT-literacy has been actively promoted in Malaysian schools by various agencies of the 
Malaysian Ministry of Education. The Ministry has also made it compulsory for all trainee teachers to be 
exposed to ICT tools, and by implication, the use of ICT-literacy in their pedagogical strategies (Chan 
2002b). 
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The project team who was responsible for the ICT-implementation in Malaysian schools has listed three 
abilities expected of Malaysian students. This abilities include competencies to use ICT tools and ICT 
sources to: 1) collect, analyse, process, and present information; 2) support meaningful learning in various 
contexts; and 3) prepare students for employment (Smart School Project Team 1997). The competencies 
listed by the team coincide with the definition of technology literacy described by NAE and NRC; which 
called for an understanding of technology at a level that enable effective functioning in a modern 
technological society (NAE & NRC 2006). In focusing on improving school children’s ICT-literacy, it is 
possible that current Malaysian trainee teachers may not be adequately prepared to teach under this new 
approach. 

In spite of this, ICT tools must never be mistaken as the mechanism that we learn ‘from’, but rather as the 
tool that we learn ‘with’. By having ICT tools such as computers, for example, they should be regarded as a 
learning aid rather than a learning point. Yet, in looking at the current Malaysian school scenario, Chan 
(2002a) identifies a serious gap between what is being understood by the teachers and what they actually 
practice. Most teachers prefer to use ICT to enhance rather than transform their current curriculum. Rather 
than changing their old teaching module to include ICT in their teaching activities, most teachers felt that it 
would be easier to use ICT to accommodate and enhance their existing/older teaching modules. For example: 
instead of using the overhead projector (OHP), the teachers would input their teaching notes into a digital 
presentation application (for example MS Powerpoint); or instead of manually creating a class timetable, the 
teachers are now using spreadsheets. It is proposed that these teachers may not have the initiative or time to 
explore other aspects of ICT that they could use to improve their teaching activities (BECTA lune 2004). 

In 2004, a group from the National Academy of Engineering (NAE) and the National Research Council 
(NRC) conducted a study to determine the most viable approach to assessing technological literacy in the US 
for K-12 students, K-12 teachers, and out-of-school adults. The report found that there was very little 
information available on the technological literacy of teachers (NAE & NRC 2006). Although many school 
children have sophisticated technological capabilities, they cannot be fully technologically literate unless 
their teachers are. There is an urgent need for an in-depth study on this topic and development of a suitable 
task-based (technological capability) assessment instrument. 


3. THE NEED FOR A DIFFERENT INSTRUMENT TO ASSESS TRAINEE 
TEACHERS’ ICT-LITERACY 

Several studies have been conducted on students (Katz 2007; Russell & Finger 2007), trainee teachers, and 
in-service teachers (Luke 2001; Knezek & Christensen 2002; Jamieson-Proctor, Burnett, Finger & Watson 
2006). Most of them involve participants’ perceptions and attitudes on their preparedness to integrate ICT as 
tools or teaching ICT as the class subject. 

The quality of the research design in ICT-literacy studies is also an issue because too many research 
studies applied the self-assessment methodologies (see Jamieson-Proctor, Burnett, Finger & Watson 2006; 
Markauskaite 2007; Ball & Levy 2008). Computer self-efficacy involves a belief of one’s capability to use a 
computer. Markauskaite’ s (2007) study for example, utilised the self-efficacy theory in her questionnaire 
design. Each test-item in her questionnaire started with the phrase “I believe I have the capability...” and was 
measured using a six-point Likert scale. 

Other studies applied a combination of both self-efficacy methodology and a more hands-on evaluation of 
the participants ICT-literacy. One such example was in Wong (2002) study. In her study, she developed an 
IT preparedness assessment instrument, which measured teachers’ preparedness in using ICT using three 
different measures: a self-efficacy instrument to evaluate their attitude towards using ICT; a hands-on 
instrument to assess their ICT skills; and ICT-knowledge exam-based questions that consists of 25 multiple 
choice questions which tested the teachers knowledge on ICT and computers. Nonetheless, the hands-on 
instrument lacks flexibility and does not encourage critical and analytical thinking, as it gives the teachers 
step-by-step instructions of what was to be performed, and the exam-based questions may not represent the 
teachers’ actual ICT knowledge as the multiple choice questions may allow guessing. 

To cope with today’s technological demands, people need to acquire more than just the basic ICT skills 
and knowledge. They need to know how to use their acquired knowledge and skills by: thinking critically; 
applying knowledge to new situations; analysing information; generating new ideas; communicating; 
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collaborating; solving problems; and making decisions. These skills can provide both flexibility and security. 
People who can learn new information, are able to use software programs and conceive new ways of doing 
things, have much better prospects than those who cannot (Partnership for the 21st Century Skills 2002). 

The lack of ability to think critically and analytically, and also to make decisions is apparent in Malaysian 
school students. Earlier studies have shown that students’ critical and analytical thinking abilities in Malaysia 
were between below satisfactory and fair (Zaharah 1995; Azlan 1997; Razali 1999). 

It is further noted that Malaysian teachers may also lack the ability to teach these skills or are less 
prepared to teach by incorporating these skills in teaching and learning activities (Rajendran 2001; Rosnani 
2002). Teachers may understand the importance of teaching critical thinking to students, yet some appear to 
not have the necessary instructional strategies to teach it (Rosnani 2002). These issues corroborate this 
study’s need for an enhanced ICT-literacy TBA instrument that is flexible; not based on self-assessment; and 
includes test-items that test cognitive skills. The tasks for the ICT-literacy TBA instrument should focus on 
familiar/normal, computer-based activities for a classroom environment that teachers usually find in their 
schools. 

A task-based assessment method, allows the participants the freedom to complete the task in any way 
they wish, as long as the task requirement is fulfilled. It means that, if the task asks for an appropriate 
learning aid to be created which includes an image and a video, the participant is free to use whatever 
computer applications that they feel comfortable to use, to edit the pictures, create videos and other digital 
learning aids. As long as the task requirement is fulfilled, the task is considered complete. Task-based tests 
allow the participant to demonstrate acquired knowledge capacity and an ability to correlate tasks with the 
theories or concepts learned previously. Instead of judging knowledge acquisition through a series of 
multiple choice selections or self-evaluation, task-based assessment forces participants to place knowledge 
into a context that can be understood and explained (Teachnology Inc 2011). Task-based assessment also 
allows the participants to show what they know, instead of just telling what they think they know. It is 
considered as the best method for this new ICT-literacy assessment instrument as it shows the participant’s 
actual ICT ability. 


4. THE METHODOLOGY 

This study was conducted in three phases: Phase-1 preliminary review; Phase-2 expert judgement on ICT- 
literacy indicators; and Phase-3 instrument validation and testing. In Phase- 1, review of the literature was 
conducted, where it involves drawing on the existing literature on ICT-literacy; ICT-literacy standards; 
existing ICT-literacy assessment instruments; and the Malaysian Smart School ( MSS) requirements. Eleven 
ICT-literacy indicators were identified in this first research phase. In Phase-2, the identified ICT-literacy 
indicators were evaluated by a specially chosen panel of experts (PoE). Two Delphi interactions were then 
conducted where the first was to evaluate the ICT indicators, and the second was to validate the draft 
ICT-literacy TBA instrument. In Phase-3, the draft ICT-literacy TBA instrument was validated and tested 
through two pilot tests, and finally the instrument was tested on a larger number of participants for its final 
instrument Uial. 

The Delphi technique and the Rasch IRT model were applied for the qualitative and quantitative parts of 
this study, respectively. The Delphi technique was chosen as the most suitable philosophical/research 
approach for conducting the expert judgement phase (Phase-2) of this study. 

For the quantitative part of this study, the Rasch IRT model was employed. The Rasch IRT model is 
assumed to be invariant across different groups within a research population and across populations 
(Hambleton & Murphy 1991; Swaminathan 1999). It requires that both the test-items and participants 
conform to the Rasch IRT model before claims regarding the presence of skill or ability can be considered 
valid. Therefore, under this Rasch IRT model, mis-fitting responses require a reason for the misfit, and may 
be excluded from the data set if they fail to address the expected skill or ability. The Rasch IRT model also 
provides a way of measuring the quality of the test-items by confirming their suitability for participants and 
how well they measured participants’ abilities (Wu & Adams 2007). 

The Rasch IRT model generally utilises the response pattern. It assumes that participants with a low 
attribute have little chance of guessing the correct answer and participants who achieve a high attribute will 
almost certainly choose the correct answer (Nunnally & Bernstein 1994). Central to idea of the Rasch IRT 
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model is the probability principle. A person’s response to a particular test-item is never certain. It is always 
influenced by human error. Thus, a probabilistic approach must be employed. In the Rasch IRT model, 
probabilities are introduced through consideration of the odds that a person would give a correct response to 
a test-item. This is known as logit. The logit is a mathematical model that converts both difficulty and ability 
into the same units. The logit is a log e of the odds of a correct response given. The Rasch dichotomous model 
equation used in the Quest analysis describes the probability of observing a specific score X n \ as (Adams & 
Khoo 1996): 




sxp(x K tWti(0*- s t0) 

1 4- exp (ar nt *F tl 0?„ - 5 t J) 


Where X n { is person n s response to item i, f! n is the ability of person /?, W, \ is the score assigned to one 
step in item i, and du is the difficulty of the one step in item i. 

The validation and testing process showed that the ICT-literacy TBA instrument is valid and reliable when 
tested on its intended participants, and the instrument is ready. The instrument provides information with 
regard to each participant’s area of weakness in ICT. 


5. THE PARTICIPANTS 

There were two categories of participants for this study: 1) the qualitative study participants (the PoE 
members); and 2) the quantitative study participants (the Malaysian trainee teachers). 

5.1 Qualitative Study Participants - PoE Members 

There were seven PoE participants for the qualitative study and they were selected based on their educational 
and occupational backgrounds. The PoE members were three academics in the field of educational 
technology, and one officer from the Malaysian Ministry of Education, and three teachers from the MSS. 

5.2 Quantitative Study Participants - Malaysian Trainee Teachers 

One of Malaysia’s public universities, the Sultan Idris Education University (UPSI), was chosen for the 
location to conduct the research. This is the only teachers’ university in Malaysia with the sole purpose of 
training pre-service teachers. For this study, the participants were undergraduate students who were currently 
enrolled in a Bachelor of Education degree at UPSI. The university has eight faculties offering both 
undergraduate and postgraduate degrees. The students are generally between 20 to 30 years old, and of mixed 
ethnicity (Malay, Chinese, Indian, Indigenous and others). Participants from the same semester across eight 
different faculties were invited to take part. 

Different participants were used for each of the three activities (pilot test-1, pilot test-2 and final 
instrument testing) in this third phase of the ICT-literacy TBA instrument development. During the pilot 
testing-1 session, 16 trainee teachers were randomly chosen and were willing to participate. Twenty (out of 
the 50 invited) trainee teachers from the Faculty of Business and Economics participated in pilot testing-2. 
For the final instrument testing process, 148 (out of the 382 invited) trainee teachers from the semester-four 
batch, representing all faculties in the university, agreed to participate. 


6. THE FINDINGS 

There were three parts of findings which was based on the different phases of the study. As previously 
explained. Phase- 1 involved identifying suitable ICT-literacy indicators to be used to evaluate trainee 
teachers level of ICT-literacy. The findings from Phase-2 and Phase-3 are discussed below. 
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6.1 The Findings from Phase-2 

The seven PoE participants reached consensus after two Delphi-rounds have been conducted. They agreed 
that there were twelve ICT-literacy indicators that should be used to evaluate trainee teachers’ level of ICT- 
literacy. The twelve indicators were: 


Table 1. List of finalised ICT-literacy indicators 


Identified ICT-literacy indicators : Finalised 

1 . 

Navigation and search; 

7. Manage; 

2. 

Production and analysis; 

8. Create; 

3. 

Plan / define; 

9. Communicate /collaborate; 

4. 

Access; 

10. Reflect; 

5. 

Integrate; 

11. Understanding and handling ICT tools. 

6. 

Evaluate; 

12. Assess 


The PoE participants also validated the draft ICT-literacy TBA instrument where the ICT-literacy 
indicators was transformed into ICT-based tasks which were common for teachers. For evaluation purpose, 
each of these tasks was divided into a few test-items. 


6.2 The Findings from Phase-3 


The draft ICT-literacy TBA instrument underwent a continual validation and testing process in a series of 
pilot studies, before it was tested on a larger population of trainee teachers in the final instrument testing 
process. The Rasch IRT model and the Quest interactive test item analysis system (Adams & Khoo 1996) 
was implemented as the data analysis tool for these performance data validation and testing. 

For the final instrument testing, all 148 participants completed all six tasks in the ICT-literacy TBA 
instrument. The responses for the ICT-literacy TBA instrument were coded by the researcher into an 
electronic data file using Microsoft Excel. A text file was then prepared for the Quest analysis (Adams & 
Khoo 1996). 

The Quest analysis produces a Kidmap of an individual participant’s performance in terms of depicting 
their correct and incorrect response patterns, according to the Rasch IRT model’s expectations. The Kidmap 
locates each test-item on a vertical scale according to its difficulty from easiest to hardest and then separates 
the test-items horizontally (left or right), according to whether the participant answered them correctly or not. 
Importantly the map locates the participant’s ability on the same vertical scale. 



Figure 1. Kidmap - showing an individual’s performance 
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Based on the Kidmap findings, a graph of achievement summary of all the candidates was produced using 
the data. Figure 2 showed the level of the participants ICT literacy ability based on their achievement for 
each test-item. For each test-item, the achievement were grouped into four level of competency: 1) Easier 
Achieved (EA), 2) Harder Achieved (HA), 3) Easier Not Achieved (ENA), and 4) Harder Not Achieved 
(HNA). 



This group of participants proposed that test-item 5.2 (using basic spreadsheet formula to assess student 
grade), test-item 11 (reflect/judge credible website) and test-item 17.4 (knowing how to evaluate credible 
webs) were the hardest test-item with EA value less than 10% of total participants. This finding demonstrates 
that this group of participants lack the skills involving the use of a spreadsheet application and also the skills 
and knowledge needed that would enable them to identify and judge credible websites. These skill and 
knowledge is exceedingly important for trainee teachers who are going to be teaching in a Smart School 
environment. Definitely we do not want the teachers to be teaching the school children with resources which 
are unreliable or hearsay. 

Aside from that, the findings also indicated that the learning domain tested below were easy for this group 
of participants, where none of the participants scored these items as HA or HNA (Table 2). 


Table 2. Learning domains with no Harder Achieved or Harder Not Achieved score 


Item no 

Learning domain tested 

1 

manage 

3 

basic ICT capabilities 

4 

basic ICT capabilities 

5.1 

assess 

10 

Internet navigation & search 

18.1 

perform analysis & producing appropriate document 

18.2 

perform analysis & producing appropriate document 

19 

basic ICT capabilities 


Figure 2 also showed more than 10% of the participants scored ENA on five test-items. The learning 
domains were: 
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Table 3. Learning domains with Easier Not Achieve score 


Item no 

Learning domain tested 

3 

basic ICT capabilities 

6 

perform analysis & producing appropriate document 

7 

plan & define 

8 

access 

9 

manage 


Rasch model suggested that these learning domains (Table 3) should be easy to the participants. However, 
more than 10% of them were unable to correctly complete the tasks. The reason might be because of 
carelessness, or might even be because of their over confident attitude they misread the task requirement and 
wrongly executed the tasks. It might also be because the already had the knowledge but they had forgotten 
how to execute them, due to uncommon use of the task in the participants’ daily work. Whatever the reasons, 
the participants needed to revisit this group of tasks immediately and resolve the problem. 


7. CONCLUSION 

Based on this findings, the ICT-literacy TBA instrument identified that this group of participants suggested 
that the assessing information; integrating information from multiple sources; and evaluating the information 
that they have, as the hardest ICT-literacy skills. These skills however, were among the most important ICT- 
based skills that a teacher must have. 

The ability to pin-point to the exact ICT-literacy ability that the trainee teachers lack might help them and 
their institution of higher learning in providing necessary intervention or additional skills to these trainee 
teachers. Their training can target their specific area of weakness, and redundancy in training information can 
be avoided. This in turn can save a lot of trainer’s time and cost. 
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